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Abstract 

To  effectively  study  team  variables  as  they  impact  performance  in  a  particular  domain,  it  is 
possible  to  develop  medium  fidelity  simulations  that  abstract  some  details  of  the  performance 
environment  while  maintaining  others.  This  paper  reports  the  results  of  a  successful  effort  to 
create  a  synthetic  task  environment  that  captures  key  elements  of  a  team  task,  based  on  Cognitive 
Task  Analysis  of  the  important  features  of  the  task  from  a  teamwork  and  cognition  viewpoint. 
We  studied  the  performance  of  AWACS  Weapon  Director  (WD)  teams  and,  based  on  the  CTA 
data  collected  and  insights  from  contemporary  team  theory,  adapted  an  existing  base  simulation 
to  mimic  many  of  the  crucial  details  of  the  task  and  its  teamwork  demands.  The  DDD  simulator 
is  a  unique  software  tool  set  and  computer  system  developed  to  study  issues  of  distributed 
situation  assessment  and  resource  allocation  in  a  dynamic  team  environment.  As  of  this  writing, 
we  have  we  have  run  collected  data  using  the  DDD  AWACS  task  and  knowledgeable  cadets 
from  the  US  Air  Force  Academy.  We  are  in  the  process  of  demonstrating  that  it  is  possible  to 
strike  a  balance  between  highly  complex,  large-scale,  high  fidelity  simulations  on  the  one  hand, 
and  over-controlled,  overly-simple  laboratory  research  tasks  on  the  other.  Results  of  our  research 
will  inform  team  theory,  system  and  organizational  design,  and  continued  research  in  the 
laboratory  and  field. 1 


1  The  work  reported  herein  was  partly  sponsored  by  the  Air  Force  Research  Lab  under  Contract  No.  F41624-98-C- 
6010,  Phase  II  SBIR. 
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1  Problem  Description  and  Relevance  to  C2 


The  complexity  of  team  interactions  and  the  environments  in  which  they  perform  makes  it 
difficult  to  empirically  study  teams  in  controlled  experiments.  Particularly  when  the  teams  we 
are  interested  in  studying  typically  perform  in  rich  environments  with  many  naturally  occurring 
performance-effecting  variables,  it  can  be  difficult  to  separate  the  performance  effects  of 
individual  aspects  from  the  tangle  of  interacting  factors  in  the  environment.  To  offset  some  of 
these  difficulties,  many  researchers  have  turned  to  simulation  technologies  in  an  effort  to  create 
enough  realism  to  stress  expert  teams,  yet  control  enough  environmental  noise  to  collect  clean 
team-performance  results.  In  the  military,  these  simulations  are  often  high-fidelity  "copies"  of 
actual  field  systems,  backed  by  network-coupled  control  stations  and  semi-automated  force 
models  to  provide  troops  with  very  realistic,  full  immersion  performance  experiences.  These 
simulations  often  end  up  being  as  complicated  as  the  environments  they  were  meant  to  model. 

To  effectively  study  team  variables  as  they  impact  performance  in  a  particular  domain,  it  is 
possible  to  develop  medium  fidelity  simulations  that  abstract  some  details  of  the  performance 
environment  while  maintaining  others.  To  create  a  reasonable  analog  of  any  team  performance 
environment,  however,  requires  very  careful  analysis  of  the  task  and  environment  to  insure  that 
the  variables  that  might  have  impacts  on  team  performance,  are  preserved  while  others,  that  are 
not  likely  to  influence  team  performance  are  abstracted  out  of  the  simulation.  Design  of  this  type 
requires  a  very  thorough  understanding  of  the  task  and  environment.  It  also  requires  that 
information  about  the  task  be  interpreted  through  the  lens  of  team-performance  theory  and 
models.  These  data  and  theory,  together,  can  be  used  to  decide  which  among  the  many  possible 
variables  that  might  be  modeled  are  the  important  variables  required  to  capture  the  teamwork  and 
coordination  demands  of  the  task  under  consideration. 

Cognitive  Task  Analysis  (CTA)  is  an  effective  method  for  eliciting  information  about  the 
decisions  made  and  the  information  used  by  experts  in  performing  their  jobs  (Gordon  &  Gill, 
1997).  While  it  seems  obvious  that  CTA  results  should  be  helpful  in  creating  simulation-based 
task  environments  that  capture  key  cognitive  elements  of  real-world  tasks,  the  bridge  between 
CTA  and  simulation  has  not  been  well  established.  This  paper  reports  the  results  of  a  successful 
effort  to  create  a  synthetic  task  environment  that  captures  key  elements  of  a  team  task,  based  on 
Cognitive  Task  Analysis  of  the  important  features  of  the  task  from  a  teamwork  and  cognition 
viewpoint. 

The  domain  of  application  is  AWACS  command  and  control.  We  studied  the  performance  of 
AW  ACS  Weapons  Directors  (WD)  teams  and,  based  on  the  CTA  data  collected  and  insights 
from  contemporary  team  theory,  adapted  an  existing  base  simulation  to  capture  many  of  the 
crucial  details  of  the  task  and  its  teamwork  demands.  Essentially,  AWACS  operators  are  air- 
traffic  controllers  in  the  sky:  They  direct  blue  fighters  to  do  such  things  as  intercept  targets, 
refuel  with  airborne  tankers,  run  strike  missions,  and  return  to  base.  AWACS  teams  typically 
consist  of  three  or  more  WDs  supervised  by  a  Senior  Director  (SD).  The  SD  is  responsible  for 
configuring  the  weapons  team  based  on  the  mission,  reporting  any  problems  to  individuals 
outside  of  the  section,  taking  over  for  overloaded,  overwhelmed  WDs,  and  performing  other 
important  missions  within  the  mission  (i.e.  search  and  rescue  efforts). 

WDs  must  coordinate  as  a  team  while  simultaneously  monitoring  and  acting  on  incoming 
information  in  a  very  dynamic  mission  environment.  They  must  coordinate  resources  (including 
knowledge,  assets  under  their  control,  and  information)  and  must  maintain  an  accurate  mental 


model  of  a  rapidly  changing  world.  We  have  shown  that  it  is  possible  to  capture  many  of  these 
demands  in  a  medium  fidelity  simulator,  and  we  are  currently  in  the  process  of  validating  this 
simulation  to  show  that  the  data  and  insights  it  will  provide  as  a  lower  cost,  higher-control 
research  tool  can  generalize  to  higher  fidelity  simulation  and  to  the  field. 

2  Approach 

The  goal  of  our  work  has  been  to  develop  and  validate  a  medium  fidelity  simulation  of  AWACS 
command  and  control  using  a  process  of  focussed  CTA  of  the  domain,  model-grounded 
conversion  of  CTA  data  into  a  reasonable  minimum  set  of  requirements  for  a  simulation, 
software  development,  and  finally,  validation  of  the  medium-fidelity  simulation  using  expert 
participants  and  high-fidelity  or  field-gathered  data  for  comparison.  We  are  in  the  process  of 
demonstrating  that  it  is  possible  to  strike  a  balance  between  highly  complex,  uncontrollable 
exercises,  on  the  one  hand,  and  over-controlled,  overly-simple  laboratory  research  on  the  other. 
Further,  we  are  currently  working  to  collect  data  that  can  be  directly  compared  to  data  from  a 
higher  fidelity  simulation  (C3STARS,  at  Brooks  AFB)  to  show  that  our  approach  provides  data 
that  generalize  to  the  field. 

2.1  AWACS  Command  and  Control 

During  the  first  phase  of  this  work,  CTA  interviews  were  performed  using  expert  WDs 
(MacMillan,  Serfaty,  Thordsen,  Klinger,  Cohen,  Freeman,  &  Elliott,  1997).  The  goal  of  this 
CTA  was  to  identify  the  major  cognitive  tasks  performed  by  the  WDs,  the  information  used  in 
those  tasks,  the  elements  of  team  interaction  that  are  most  central  to  the  WD  team,  and  the 
characteristics  of  team  interaction  that  distinguish  the  WD  team  from  other  teams  that  perform 
resource  allocation  tasks.  Several  key  insights  were  gathered  at  this  stage  that  allowed  us  to  adapt 
a  reasonable  medium-fidelity  simulation.  For  instance,  we  learned: 

WDs  operate  in  an  environment  in  which  teamwork  is  of  paramount  importance.  Not  only  does 
the  success  of  a  mission  often  hinge  on  the  ability  of  the  weapons  team  to  handle  the  situation, 
but  lives  hang  in  the  balance  as  well.  A  good  weapons  team  can  handle  the  difficult  fast-paced 
tasks  that  are  presented  to  them  in  every  mission.  In  contrast,  the  team  that  is  still  figuring  out 
who  talks  to  whom,  who  is  responsible  for  what,  and  how  to  pass  information  is  headed  for 
disaster. 

WDs  and  the  SDs  establish  contracts  with  one  another  prior  to  a  mission.  These  contracts  are 
quite  distinctive  to  the  AWACS  WD  team,  although  it  is  not  uncommon  for  well  coordinated 
teams  to  take  advantage  of  quiet  periods  in  order  to  make  agreements  about  how  they  will  handle 
tasks  during  more  stressful  periods  (Orasanu,  1990).  Contracts  for  the  AWACS  team  may 
include:  when  the  check-in  controller  turns  aircraft  over  to  the  Area  of  Responsibility  (AOR) 
controller,  how  the  team  will  handle  the  hand-offs  when  aircraft  go  to  tanker,  how  to  react  to  a 
search-and-rescue  mission,  etc.  If  the  team  has  not  dealt  with  these  issues  prior  to  boarding  the 
aircraft,  they  will  be  caught  off  guard  once  they  are  on  station. 

These  and  other  insights  provided  us  with  an  understanding  of  the  tasks  AWACS  teams  face,  the 
teamwork  demands  that  shape  performance,  and  the  role  of  their  individual  taskwork  in  overall 
workload.  We  coupled  these  insights  with  expert  advice  about  the  form  and  pace  of  events  WDs 
monitor  and  control  through  their  individual  computer  consoles  and  communication  equipment, 
and  developed  an  AWACS  simulation. 


2.2  Simulation  Development 

The  results  of  the  CTA  interviews  were  used  to  configure  a  team-task  simulation  testbed  that 
captures  the  most  essential  elements  of  the  AW  ACS  Weapons  team  performance.  Implications  of 
the  CTA  for  the  design  of  our  synthetic  AWACS  task  included: 

•  The  simulation  must  be  flexible  in  allowing  multiple  configurations  for  the  division  of 
responsibilities  among  team  members.  AWACS  weapons  teams  differ  in  the  way  that  they 
divide  responsibilities. 

•  Team  members  must  be  able  to  take  over  tasks  from  other,  overloaded,  team  members. 

•  Team  members  must  be  able  to  discuss  division  of  responsibilities  and  form  “contracts” 
before  they  begin  a  scenario. 

Based  on  the  results  of  the  CTA,  we  modified  an  existing  distributed  team-in-the-loop 
simulation,  the  Dynamic  Distributed  Decisionmaking  (DDD)  simulation  (Kleinman,  Pattipati, 
Luh,  and  Serfaty,  1989;  Kleinman  and  Song,  1990;  Kleinman,  Young  and  Higgins,  1996)  to 
represent  the  most  important  elements  of  the  AWACS  WD  team  task. 

The  DDD  simulator  is  a  unique  software  tool  set  and  computer  system  developed  to  study  issues 
of  distributed  situation  assessment  and  resource  allocation  in  a  dynamic  team  environment. 
Essentially,  the  DDD  is  a  team-in-the-loop  simulation  core  that  can  be  used  to  model  many 
different  environments  and  then  used  to  study  important  team-oriented  cognitive  processes  and 
tasks,  including  situation  assessment,  resource  allocation,  and  decision  making.  The  DDD 
simulator  has  been  a  major  component  of  a  team  research  program  that  has  been  underway  for 
almost  15  years(Serfaty  &  Kleinman,  1985;  Kleinman  &  Serfaty,  1989,  Serfaty,  1996). 

In  a  typical  DDD  scenario,  a  team  of  decision  makers  must  make  coordinated  decisions  based  on 
uncertain,  ambiguous,  and  sometimes  decentralized  information.  Each  team  member  has  only  a 
portion  of  the  information  and/or  resources  needed.  Figure  1  shows  a  configuration  for  a  four- 
person  team — one  leader  and  three  subordinates — with  typical  DDD  team  decision-making 
tasks.  A  team  task  such  as  the  one  illustrated  in  Figure  1  re-creates  many  of  the  cognitive 
demands  associated  with  team  decision  making  and  cooperative  work.  The  DDD  simulation  task 
is  designed  with  a  flexible  structure  that  can  be  manipulated  to  vary  a  number  of  different 
elements  of  task  complexity  (e.g.,  risk,  uncertainty,  time-pressure,  information  distribution, 
communication  structure.)  The  DDD  software  provides  real-time  control,  on-line  data 
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Figure  1 .  Typical  DDD  Configuration 


collection,  an  interactive  display/interface.  In  the  current  context,  we  demonstrated  that  the 
DDD,  with  modest  changes,  could  represent  the  team  tasks  of  AW  ACS  Weapons  Directors.  We 
used  CTA  data  and  expert  advice  from  AF  personnel  to  develop  a  medium  fidelity  analog  of  the 
Weapons  Director  console.  The  resulting  task  supports  an  n-player  game  of  WDs  monitoring  and 
controlling  resources  and  tasks  in  an  airspace.  The  simulation  provides  total  control  over 
scenario  design,  team  organization,  and  data  collection.  As  a  next  step,  we  are  validating  this 
simulation  and  developing  a  team-performance  tools  package  that  includes  empirically  tested 
measures,  models,  results,  and  specific  guidelines  for  effective  team-support  interventions. 


Figure  2.  DDD  AWACS  Display 


3  Results 

The  resulting  AWACS-DDD  simulation  captures  the  key  variables  needed  to  study  AWACS 
command  and  control  teams  in  a  more  controllable,  faster  turnaround  environment  with  enough 
richness  to  tax  even  expert  players,  but  enough  control  to  support  targeted  research.  The  current 
version  of  AWACS-DDD  captures  many  key  detail  found  in  the  actual  AWACS  Command  and 
Control  team  environment.  We  have  simplified  some  aspects  of  the  task  while  including  other 
aspects  that  theory  predicts  will  have  maximum  impacts  on  team  performance.  Examples  of 
details  that  we  have  captured  include: 

•  different  types/classes  of  hostile  aircraft, 

•  different  types  classes  of  friendly  assets  with  different  characteristics  such  as  endurance, 

•  sensors  on-board  friendly  aircraft  that  may  obtain  information  not  available  to  the  AWACS 
directly, 

•  ability  for  an  aircraft  to  pursue  a  track  (using  an  intercept  geometry), 

•  modeling  a  sensor  platform  (e.g.,  AWACS), 

•  introducing  hostiles  at  prescribed  times  to  fly  on  user-selected  trajectories, 

•  hostiles  that  can  attack  the  team’s  assets  as  well  as  defended  areas, 

•  friendly  air  bases  from  which  aircraft  are  launched  and  can  return  for  resupply, 


•  ground  targets  (for  a  strike  package), 

•  SAM  threats, 

•  SCUD  launchers  (detected  only  when  a  sensor  is  in  close  proximity)  capable  of  firing  a 
missile  at  a  defended  site, 

•  ability  to  assigning  specific  tasks  to  a  specific  WD  (as  might  be  done  by  the  SD), 

•  ability  to  transferring  control  of  assets  among  WDs, 

•  ability  to  modifying  team  member’s  responsibilities,  and 

•  ability  to  change  a  task’s  priority. 

An  example  display  for  the  DDD  AW  ACS  simulation  is  given  in  Figure  2.  The  large  grid  region 
that  takes  up  the  majority  of  the  left  portion  of  the  screen  is  the  map  over  which  aircraft  are 
animated  when  the  game  is  running.  This  map  is  the  region  of  the  screen  where  most  player 
actions  take  place.  In  this  image,  the  map  depicts  an  island  surrounded  by  a  defined  buffer 
boundary.  The  symbology  imposed  on  the  map  represent  friendly  and  enemy  aircraft  currently  in 
view.  Known  hostile  tracks  are  represented  in  red,  unknown  but  detected  aircraft  are  represented 
in  yellow,  and  friendly  aircraft  are  represented  in  green  with  colored  labels  indicating  player 
ownership.  Orbit-points  and  aircraft  destinations  (the  coordinates  that  a  player  instructed  aircraft 
to  move  toward)  are  indicated  by  small  red  circles.  Aircraft  are  shown  using  standard  symbols. 
The  lines  on  these  symbols  are  track  vectors  that  point  in  the  track's  current  direction  and  have  a 
length  that  increases  with  the  speed  of  the  track.  The  buttons  on  the  right  and  bottom  sides  of  the 
screen  have  been  implemented  with  virtual-action  buttons  to  mimic  the  AW  ACS  functionality 
and  a  simple  email  system  for  team  communication. 

The  next  step  in  the  research  will  be  to  use  the  DDD  AW  ACS  testbed  to  experimentally  test 
interventions  aimed  at  enhancing  team  strategies,  cognition,  and  performance,  including  shared 
displays,  teamwork  support  systems,  team  procedures  and  team  training  modules.  Results  will  be 
used  to  guide  the  design  and  training  of  future  team-based  systems.  The  goal  will  be  not  just  to 
prove  that  an  intervention  works,  but  to  understand  why  it  works  so  that  the  results  can  be 
generalized  to  other  kinds  of  teams. 

As  of  this  writing,  we  have  we  have  collected  but  not  yet  completed  analysis  of  data  from  an 
experiment  in  which  DDD  AW  ACS  scenarios  were  executed  by  knowledgeable  cadets  from  the 
US  Air  Force  Academy.  Our  initial  research  was  focused  on  the  organization  and  distribution  of 
responsibilities  in  the  AW  ACS  team  and  the  impacts  of  changing  workload  on  performance 
under  different  team  architectures.  Our  preliminary  results  are  encouraging.  The  task  was 
extremely  engaging,  and  produced  an  engrossing  team  experience.  We  collected  embedded 
performance  measures,  survey  data  and  observational  data  to  provide  a  complete  picture  of  team 
process,  performance,  and  outcome.  These  preliminary  data  will  provide  a  baseline  for  future 
experiments  as  well  as  insights  into  AW  ACS  team  function. 
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